Method and system to provide a computer-modified visualization of the desired face of a person

ABSTRACT

A data set of visuals of faces and extracted face property data are generated and linked to face characteristics data provided by a representative set of humans that rate the visuals of these faces with respect to their face characteristics. Further face property data of these visuals of faces is extracted and together with the generated data set used to train an artificial intelligence. The artificial intelligence is used to analyse a visual of the person&#39;s face and generate a data set of modifications based on a selected desired characteristic(s) and modifications achievable by at least one cosmetic and/or medical treatment. The visual of the face of the person is modified based on the data set of modifications and the computer-modified visual of the desired face of the person with the modification of the face achievable by the least one proposed cosmetic and/or medical treatment is generated and displayed.

CROSS REFERENCE TO RELATED APPLICATION

This application claims benefit of European Patent Application SerialNo. 19215134.8, filed 11 Dec. 2019 and which application is incorporatedherein by reference. To the extent appropriate, a claim of priority ismade to the above disclosed application.

BACKGROUND

The present invention relates to a method and system to provide acomputer-modified visualization of a desired face of a person whoconsiders undergoing a minimally invasive and/or invasive cosmeticand/or medical treatment to improve the person's appearance. There is ageneral wish to optimize the own appearance. The face is one of the mainareas of the body relevant for this appearance. There are many differenttreatments known to change the facial appearance, expression and/ormorphology, e.g. reduction of wrinkles in the skin of the face ormodification of the cheekbones. A person interested in such a changetypically makes an appointment with a beautician, dermatologist,physician, specialist certified to do facial modifications, or plasticsurgeon to get information about possible treatments. In a first stepthe specialist mentioned above inspects the different regions of theface and based on their personal knowledge and skills proposes medicaland/or cosmetic treatments that could provide the desired change of theappearance. The problem of this way of working is that it is difficultfor the person to understand the possible visual effects and/or effectson the first impression (i.e., how the face is perceived by others) ofthe treatments the specialist proposes.

LIAO YANBING ET AL: “Deep Rank Learning for Facial Attractiveness”, 20174^(TH) IAPR ASIAN CONFERENCE ON PATTERN RECOGNITION (ACPR), IEEE, 26Nov. 2017 (2017-11-26), pages 565-570, XP033475316 discloses a DeepConvolutional Neuronal Network and artificial intelligence for fullyautomatic facial beauty assessment and ranking. A “HOTorNOT” database of1.885 female face images collected from a popular social/dating websitewas used to train the artificial intelligence how to predict facialattractiveness of female faces and rank them.

MESSER U ET AL: “Predicting Social Perception from Faces: A DeepLearning Approach”, ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLINLIBRARY CORNELL UNIVERSITY ITHACA, N.Y. 14853, 29 Jun. 2019(2019-06-29), XP081385771 discloses a Deep Convolutional NeuronalNetwork and artificial intelligence to predict human perceiver'simpression of the characteristics “warmth” and “competence” based on avisual representation of a face of a person. The artificial intelligenceextracts features or face property data from regions of face images andwas trained with a 10K Adults Face database where human raters rated thecharacteristics “warmth” and “competence” of these faces to generateface characteristics. Heat maps were used to identify regions of facesrelevant for human perceivers.

WO 2015/017687A2 discloses a method and system that enables a user totake a picture of their face with a device and send it to a computerserver. The server uses a facial recognition software to identifyanatomical features of the face and the user selects the anatomical areaof his face he wants to improve. In a next step, the server determinesthe level of “severity” of a defect in the selected anatomical area.After that, the person has to select a medical or cosmetic treatmentfrom a list of possible treatments and the server determines the likelytreatment outcome based on data of clinical studies for this selectedmedical or cosmetic treatment. Finally, the server generates a modifiedpicture of the user's face with the likely outcome of the selectedmedical or cosmetic treatment for the selected anatomical area, whichpicture is displayed at the user's device next to the original picturetaken. The disclosed method provides the disadvantage that a user isleft alone both with the selection of facial regions and with theselection of possible medical or cosmetic treatments which might lead toan overall less attractive appearance of the user after one or moretreatments. There is a need for technical means to solve these technicalselection problems.

SUMMARY

These problems are solved with a method that comprises the followingsteps:

-   Generate a data set of visuals of faces and extracted face property    data thereof linked to face characteristics data provided by a    representative set of humans that rate the visuals of these faces    about their face characteristics and store the data set in a    database;-   Extract further face property data of these visuals of faces and use    these extracted face property data together with the generated data    set for training of an artificial intelligence to enable the    artificial intelligence to provide an automated rating of the    characteristics of the visuals of faces;-   Generate a data set of visual modifications of a face achievable by    cosmetic and/or medical treatments and store the data set in a    database;-   Take a standardized visual of the face of the person;-   Input at least one desired characteristic of the face of the person    to be changed;-   Use the artificial intelligence to analyse the visual of the    person's face and to generate a data set of modifications based on    the selected desired characteristic(s) and modifications achievable    by at least one cosmetic and/or medical treatment;-   Modify the visual of the face of the person based on the data set of    modifications and generate the computer-modified visual of the    desired face of the person with the modification of the face    achievable by at least one cosmetic and/or medical treatment;-   Display the computer-modified visual of the desired face of the    person.

This inventive method, system and/or computer program uses a completedifferent concept and technique to enable an informed decision for auser how to change the facial appearance, expression and/or morphology.The user may be for example, a beautician, dermatologist, physician,specialist certified to do facial modifications or plastic surgeon, aswell as a person interested in a change of their own appearance.

The invention is based on the finding that when aiming to change aperson's appearance it is only a secondary goal to e.g. selectivelyreduce frown lines or to increase the volume of the lips, since personscategorize the appearance of others in the course of forming a firstimpression in a more complex way and as a whole. Characteristicsattributed to a person when making a first impression are, for example,attractiveness, healthiness, youthfulness, tiredness, sadness,friendliness, dominance, competence, likability or trustworthiness justto name some of these. The new concept and technique enables a user toselect at least one characteristic she/he wants to change. A change in acharacteristic attributed by others during forming the first impressioncan be achieved in both ways, i.e. increasing a characteristic which isperceived as positive or decreasing a characteristic which is perceivedas negative. For example, a person might wish to appear less tired. Inanother example, a person might wish to appear more friendly. In afurther example, a person might wish to appear more attractive.

In a first step of the inventive method, a new data set of visuals offaces is compiled. This includes images and videos of people's faces aswell as computer-generated images and 3D models of artificial faces. Ina pre-processing step for the deep learning algorithm, this data set isnormalized via transformations for face alignment, cropping andresizing. Facial landmark detection of face properties first and thenmore detailed like skin texture analysis are performed in order toproperly align face data. The data set of visuals is generated andimproved with human assessments and/or modifications of such faces totrain a deep learning based application and/or artificial intelligenceand/or software. The quality of this application is further refined inan iterative process using data generated by previous versions ofitself. Reference is made to above cited scientific articles as state ofthe art documents that describe the structure and model architecture ofsuch an artificial intelligence.

According to deep learning principles, a new technique is used togenerate a data set of modifications that comprise complex informationabout all modifications of the face needed to change the user'scharacteristics in the desired direction. The modifications might affectthe facial appearance, expression and/or morphology. Such modificationsof the face may comprise, for example, height of the eyebrows, fullnessof the cheeks, width and height of the chin, volume of the lips, volumeof the zygoma, depth of the marionette lines, straightness of the jawline, depth of the glabellar lines, periorbital hollowness and skintightness. Technically, these changes can be obtained by minimallyinvasive and/or invasive cosmetic and/or medical treatments, as has beenproven by clinical studies or treatments performed in the past. Thesetreatments include application of dermal fillers, Botulinum toxin,threads, implants, surgery, laser treatments, autologous fattransplantation, and skin resurfacing treatments amongst others.

In a further step, visual data of the face of the person (e.g. photos orvideos) are obtained. Together with a selection of at least onecharacteristic to be changed, the data set of visuals of faces aretransferred to a server. The additional input of age and/or genderand/or ethnicity data of the person is possible as well.

In a final step of the method, a deep learning basedapplication/artificial intelligence processed by the server is used tomodify the visual data of the user's face according to the user'sselected change of one or more characteristics. The artificialintelligence can also be used to optionally describe and/or rate thecharacteristics based on the original visual data of the person's face.

The computer-modified visualization of the desired face can be shown ona display next to the original visual data of the user to show thepossible change. This change may be obtained by using one or acombination of different invasive and/or minimally invasive cosmeticand/or medical treatments that in sum modify the user's face towards thedesired change of the selected characteristic(s). Optionally, a proposalfor the necessary treatments to achieve the desired face can be given.

This new method provides the major advantage that it visualizes theeffect of changes of the facial appearance, expression and/or morphologyon the characteristics attributed to a person when making a firstimpression. The user has the option to choose at least one specificcharacteristic he/she wishes to change (e.g. reduce or improve).Subsequently, the needed changes of the facial appearance, expressionand/or morphology, which are necessary to reach the desired effect (i.e.the desired face), are visualized.

Methods according to the state of the art visualize only changes ofisolated regions of the face. In the context of characteristicsattributed to a person when making a first impression the face of theuser might be changed in a not desired way. The main aim of abusinessman or politician might be to improve his appearance towardsbeing perceived as a competent person while at the same time he wishesto appear younger. However, the selected changes in the facialcharacteristics to look younger might result in a less competentappearance contradicting his professional needs.

The inventors found that taking into consideration one or morecharacteristics attributed to a person when making a first impression isessential for optimizing the choice and/or selection of individualizeddifferent invasive and/or minimally invasive cosmetic and/or medicaltreatments in order to reach the desired modification of the face. Oneof the requirements for applying this principle is the ability toprocess large data sets in a novel, innovative, fast and efficient wayvia using an artificial intelligence.

Therefore, the invention solves the technical problem of prior art toprocess data in order to analyse, visualize and predict a person's faceand its perception by others according to characteristics attributed tothe person when making a first impression, by a combination of stepslisted above.

BRIEF DESCRIPTION OF THE DRAWINGS

These and further advantageous embodiments of the invention will beexplained based on the following description and the accompanyingdrawings.

FIG. 1 shows a system to display a computer-modified visualization of adesired face of a person.

FIG. 2 shows a mobile device of the system with a picture of the face ofthe person.

FIG. 3 shows in which regions the face of the person is divided forfurther analyses.

FIG. 4 shows a description and rating of the characteristics based onthe original visual data of the person's face.

FIG. 5 shows how characteristics may be selected by the user.

FIG. 6 shows the face of the person with a data set of modificationsoverlaid.

FIG. 7 shows a comparison of the original picture of the face of theperson with the computer-modified visualization of the desired face ofthe person.

FIG. 8 shows a recommendation which treatments to use.

FIG. 9 shows a line drawing of a face with regions of the face marked tobe treated to increase the characteristic “dominant” attributed to aperson when making a first impression.

FIG. 10 shows a picture of a face with regions of the face marked to betreated to increase the characteristic “dominant” attributed to theperson when making a first impression.

FIG. 11 shows a line drawing of a face with regions of the face markedto be treated to increase the characteristic “competence” attributed toa person when making a first impression.

FIG. 12 shows a table with examples of invasive and/or minimallyinvasive cosmetic and medical treatments to achieve changes of desiredcharacteristics of a person's face by actions in particular regions ofthe face of the person.

FIG. 13 shows a visual of a face of a person before and after theperformance of the recommended treatments.

DETAILED DESCRIPTION

FIG. 1 shows a system 1 to display a computer-modified visualization orvisual of a desired face of a person 2 with a mobile device 3. Themobile device 3 processes a software and in particular an App for person2, who considers undergoing an invasive and/or minimally invasivecosmetic and/or medical treatment, or for a specialist performing suchtreatment who would like to take a data-driven decision which treatmentsto choose to obtain the desired changes of the face. A camera of themobile device 3 is used to obtain the visual of the face of person 2 asstandardized visual data 4 shown in FIG. 2. Visual data 4 may representa photo or a film of the face of person 2. Standardization of the visualdata 4 may be split into instructions for person 2 and the photographerwhat to do for taking a standardized photo or film and into apost-processing of the photo or film taken. The instructions for person2 and the photographer may include one or more of the following steps:ask person 2 to take off e.g. earrings or a nose ring; ask person 2 notto smile, ask person 2 to make a neutral facial expression; ask person 2to keep head hair out of his/her face; ask person 2 to look straightinto the camera; good general condition of lightning; neutralbackground. The post-processing of the photo or film may include one ormore of the following steps: cut-out the background behind the face fromthe visual data 4; cut-out the ears of the person's face to reduce thevisual influence of e.g. earrings; cut-out clothes and other wardrobethat might influence with the face; cut-out the head hair of the person2.

System 1 comprises a remote server 5 connected via a broadband network 6or other remote connection technology with the mobile device 3. Theserver 5 processes a deep learning based application 7 and as such formsan artificial intelligence that analyses visual data 4 of the face ofperson 2 to rate one or more characteristics attributed to a person 2when making a first impression. Such face characteristics or traits mayfor example be attractiveness, healthiness, youthfulness, tiredness,sadness, friendliness, dominance, competence, likability ortrustworthiness. The deep learning based applications 7 is a computerprogram comprising instructions which, when the program is executed byremote server 5, causes remote server 5 to carry out the following stepsto provide a computer-modified visualization 13 of a desired face ofperson 2.

To enable the deep learning based application 7 to rate facecharacteristics the following steps are processed. In a first step adata set of visual data of visuals of faces and extracted face propertydata thereof linked to face characteristics data is generated. Toextract face properties conventional computer vision algorithms like alandmark detection divide the face of person 2 in regions like the chinand the jawline as shown in FIG. 3 and automatically extractscharacteristics and their location in the face. Such face property datamay for instance include the distance between the eyes or the distancebetween the eyes and the mouth and other distances to be measure todescribe a face. These face property data are stored together with thevisual data of these faces by the deep learning based application 7 in adatabase 8 of the server 5. A representative number of such visuals offaces stored as visual data in database 8 are shown on a display to arepresentative number of humans to manually rate these visuals of facesabout their characteristics. The humans may rate them with scores (e.g.from 0 to 7) for different characteristics or traits. These humanratings are stored in database 8 linked to the visual data of the facesand provide a basis information for the deep learning based application7 to rate characteristics attributed to a person 2 when making a firstimpression.

In a second step further face property data of these visuals of facesare extracted by conventional computer vision algorithms for examplelandmark detection, wrinkle detection, skin texture analysis, analysisof facial proportions. These face property data of visuals of faces areused together with the data set generated and stored in database 8 inthe first step for training of the deep learning based application 7 toenable the artificial intelligence to provide an automated rating of thecharacteristics of the visuals of faces. Reference is made to thescientific articles listed above that describe the structure and modelarchitecture of such an artificial intelligence. As a result, any visualof a face may be provided to the deep learning based application 7,which will, based on the data set stored in database 8, provide anautomated rating of the characteristics of the visuals of the face. FIG.4 shows the result of such a description and automated rating of thecharacteristics or traits of a person based on the visuals of theperson's face displayed on mobile device 3.

Server 5 furthermore comprises a database 9 with data generated in athird step based on clinical studies, case studies or other publiclyavailable information, which data comprise information about visualmodifications of a face achievable by invasive and/or minimally invasivecosmetic and medical treatments. This database 9 for instance comprisesinformation of the effectiveness of a treatment improving a wrinklescore of 3.1 to 1.5 within 3 weeks. Reference is made to prior art WO2015/017687 that discloses parameters-based clinical trial summaries,for example the investigator's rating of glabellar line severity atmaximum frown and the subject's global assessment of change inappearance of glabellar lines after a medical treatment with Botox overa time period of 120 days post-injection. This prior art documentdiscloses a database that stores data how to improve one particular areaof a face with one particular medical treatment. FIG. 12 as one furtherexample shows a table, which includes the information of database 9 withexamples of invasive and/or minimally invasive cosmetic and medicaltreatments to achieve actions or improvements in particular regions ofthe face of person 2.

After databases 8 and 9 have been setup with above described threesteps, system 1 is ready to be used to provide computer-modified visualsof a face of a person as described in the following steps of the method.

In a fourth step the camera of mobile device 3 is used to obtain thestandardized visual data 4 of the face of person 2 as described aboveand shown in FIG. 2. In a preferred optional embodiment, these visualdata 4 are sent to sever 5 and deep learning based application 7processes an automated rating of the characteristics or traits of person2 and provides the rating shown in FIG. 4 to support person 2 to decidewhich characteristic or trait he/she might want to change. In anotherless preferred embodiment person 2 makes his/her decision whichcharacteristic of his/her face to improve without the automated ratingshown in FIG. 4.

In a fifth step person 2 inputs at least one characteristic of the faceof the person to be changed based on the personal interest of person 2.The characteristic(s) input by person 2 is stored and transmitted in theform of face characteristics data. The person 2 may decide to changehis/her facial appearance, expression and/or morphology for improvingthe characteristic “competent” with input means 10 of mobile device 3 asshown in FIG. 5. This characteristic selected by person 2 is transmittedvia broadband network 6 to server 5. In another embodiment of theinvention person 2 may use another way to input the at least onecharacteristic of the face he/she is interested to change with mobilephone 3. This may be done by the selection of the App used by person 2,as there may be an App to improve the characteristic “competent” andanother App to improve the characteristic “dominant”.

In a sixth step of the method the artificial intelligence of the server5 analyses the visual of the person's face and generates a data set ofmodifications 12 based on the selected desired characteristic(s) andmodifications achievable by at least one cosmetic and/or medicaltreatment of database 9. This means that either only one or morecosmetic treatments or only one or more medical treatments or anycombination of one or more cosmetic and medical treatments of database 9may be used to modify the user's face towards the desired change of theselected desired characteristic(s). Furthermore, references to acosmetic treatment are meant to cover any invasive and/or minimallyinvasive cosmetic treatment. To achieve this technical step, the deeplearning based application 7, based on database 8, evaluates whichmodifications are needed to improve the characteristic “competent” ofperson 2 and matches these modifications needed with modificationspossible as stored in database 9. FIG. 11 shows a line drawing of a facewith regions of the face marked to be treated to increase thecharacteristic “competent” attributed to a person when making a firstimpression. In line with that, the table of FIG. 12 shows the actionsneeded to increase the characteristic “competent” attributed to person 2when making a first impression: make the chin less wide and the cheeksless full and lower the eyebrows. FIG. 12 furthermore shows the cosmeticand/or medical treatments, which can be performed by a beautician,dermatologist, physician, specialist certified to do facialmodifications, or plastic surgeon to realize these actions.

The result of a best match of modifications needed and modificationspossible are then stored in the data set of modifications 12. This dataset of modifications 12 technically describes what modifications areneeded to modify visual data 4 of the face of person 2 to show thepossible result of one or more invasive and/or minimally invasivecosmetic and/or medical treatments to improve the characteristic“competent” of person 2 in a computer-modified visual 13 of the face ofa person 2 as shown in FIG. 6. FIG. 6 shows the face of person 2 with anoverlay of arrows that indicate with regions of the face need to betreated to achieve the desired result of improved “competence”. So forinstance, the eyebrow position needs to be lifted and the volume of thejawline needs to be increased. The arrows shown are only symbolic asdata set of modifications 12 may comprise further information about theface and processing of the visual data 4 needed.

In as seventh step of the method the visual data 4 of the face of theperson 2 are modified based on the data set of modifications 12 and acomputer-modified visual 13 of the face of the person 2 with themodification of the face achievable by the at least one proposedcosmetic and/or medical treatment is generated. If for instanceartificial intelligence of the server 5 concluded that the eye bags ofperson 2 need to be tightened to improve the characteristic“attractiveness” of person 2, then the data set of modifications 12 mayinclude information to soft focus the area of lower eyelid and zygoma inthe visual data 4 of the face of person 2. Deep learning basedapplication 7 or other conventional image processing methods like imagewarping therefore processes the photo or film of person 2 to provide thecomputer-modified visual 13 of the desired face of person 2. Thisseventh step may be processed by server 5 due to its large processingpower, but could be processed by mobile device 3 as well. In thisembodiment server 5 therefore comprises picture data modification means14 to modify the visual data 4 of the face of person 2 according to thedata set of modifications 12, which modified visual data 4 of the faceof person 2 are transmitted to mobile device 3 and displayed ascomputer-modified visual 13 of the desired face of person 2 with mobiledevice 3.

In a preferred embodiment of the invention the artificial intelligenceis used to automatically identify an area with wrinkles in the visual ofthe person's face based on technologies known to a man skilled in theart. Such areas for instance may be the area of lower eyelid and zygoma.The artificial intelligence may then be used to automatically soft focusthese identified areas in case for instance the characteristic“attractiveness” of person 2 should be improved and the data set ofmodifications therefore includes such information to modify the visualof the face of the person to generate the computer-modified visual ofthe desired face of the person. In a further preferred embodiment, theartificial intelligence may also add wrinkles to the visual of a forinstance young person's face, who wants to improve the characteristic“competence” in areas where elder people use to have wrinkles.

In an eighth step of the method the computer-modified visual 13 of thedesired face of the person 2 is displayed with mobile device 3 as shownin FIG. 7. There are two preferred modes to display the visual data 4 ofthe face and the computer-modified visual 13 of the desired face ofperson 2 to enable him/her to easier see the differences. The firstpreferred mode is to use a toggle mode to alternatively show the takenstandardized visual data 4 of the face of the person 2 and thecomputer-modified visual 13 of the desired face of the person 2. Person2 just has to touch the display of mobile device 3 in a button area totoggle between the two visuals as fast as person 2 wants to see them tobetter see the differences and modifications. The second preferred modeis to use a marking mode to mark the areas of the face of the person 2modified by the data set of modifications 12 as an overlay to thedisplayed computer-modified visual 13 of the desired face of the person2 as shown in FIG. 10. Marking may be done e.g. with lines or brokenlines overlaid over the computer-modified visual 13 of the desired faceof the person 2. Both preferred modes enable person 2 to easily seethose areas of the face that would need to be treated with invasiveand/or minimally invasive cosmetic and/or medical treatments.

FIG. 13 shows the face of another person 2 before and after therecommended treatment was performed. The left photo shows person 2 whowas interested to change the appearance of his face by increasing thecharacteristics “dominance” and “competence”. The inventive method andsystem provided and displayed a computer-modified visual 13 of thedesired face similar to the right photo and provided a recommendation touse the treatment of lipofilling in particular identified regions of theface. After the recommended treatment was performed, the right photo ofFIG. 13 was taken and it turned out that the computer-modified visual 13of the desired face was nearly identical to the actual photo taken afterthe treatment. This technical solution helps substantially to makeinformed decisions about cosmetic and/or medical treatments.

It is furthermore possible to display all invasive and/or minimallyinvasive cosmetic and/or medical treatments stored in database 9 and toselect some of these invasive and/or minimally invasive cosmetic and/ormedical treatments upfront to send these together with the visual data 4of person 2 and the characteristic input by person 2 to change his/herfacial appearance to server 5. In this case, the artificial intelligenceof server 5 only uses those selected invasive and/or minimally invasivecosmetic and/or medical treatments during their search for a best matchachievable for the data set of modifications 12 in the database 9 ofvisual modifications achievable by selected invasive and/or minimallyinvasive cosmetic and/or medical treatments. This enables person 2 todecide upfront which of the invasive and/or minimally invasive cosmeticand/or medical treatments are acceptable to be used to change his/herfacial appearance and helps to streamline processing of server 5.

In case the sevenths step of above described method is processed bymobile device 3 and not on server 5, then mobile device 3 would comprisevisual data modification means 14 to modify the visual data 4 of theface of the person 2 with the received data set of modifications 12 fromserver 5 by altering certain image vectors. The computer-modified visual13 of the desired face of the person 2 is then displayed with the device3. In another embodiment of the invention image processing by visualdata modification means might be split between the server 5 and mobiledevice 3.

Deep learning based application 7 is optionally built to evaluate theage period, ethnicity and gender of person 2 in picture data 4. Thishelps to reduce data needed to be input when using the App.

System 1 furthermore enables to show those invasive and/or minimallyinvasive cosmetic and/or medical treatments 16 on the display of mobiledevice 3 that have been selected by the artificial intelligence toachieve the desired face as shown in FIG. 8. Person 2 may decide to usea filter to select only some of the shown invasive and/or minimallyinvasive cosmetic and/or medical treatments, if for instance person 2 isnot willing to undergo a surgical intervention. This selection of person2 is sent to server 5, which calculates the necessary data set ofmodifications 12, which are achievable with the reduced number ofinvasive and/or minimally invasive cosmetic and/or medical treatments toachieve the desired changes in the characteristic (e.g. attractiveness).

It is furthermore possible to input more than one characteristic withinput means of mobile device 3 as shown in FIG. 5. This provides theadvantage that person 2 is free to improve two or more characteristicsbased on his/her personal interest.

It is furthermore possible that the person that uses the App is not theperson that wants to change his/her appearance, but a person that wantsto enable an informed decision like for example, a beautician,dermatologist, physician, specialist certified to do facialmodifications or plastic surgeon.

In a further embodiment of the invention, the computer program realizedas an APP of a mobile phone may be programmed to ask the user questionslike the following: gender, age, profession, level of education, sexualorientation, religion and political orientation. These questions may beasked in the fifth step of above explained method about the user in oneembodiment and about the target group, the user is interested in, in asecond embodiment. This information may be used in the sixth step ofabove explained method, when generating the data set of modifications.The result of an analyse of the information about the user and/or thetarget group, the user is interested in, for which the user wants to berecognized as e.g. more “dominant” may be used as further input howcharacteristics of the user need to be modified. This has the advantagethat the modifications closely fit the personal needs and wishes of theuser.

In a further embodiment of the invention, an artificial intelligencesystem is used to provide an automated rating of the characteristic ofthe visuals of faces. Such artificial intelligence system may include asoftware to detect landmarks in the face of a person or any otherconventional algorithm with the ability to do so.

1. A method to provide a computer-modified visual of a desired face of aperson, wherein the method comprises the following steps: generating adata set of visuals of faces and extracted face property data thereoflinked to face characteristics data provided by a representative set ofhumans that rate the visuals of these faces about their facecharacteristics and store the data set in a database; extracting furtherface property data of these visuals of faces and use these extractedface property data together with the generated data set for training ofan artificial intelligence to enable the artificial intelligence toprovide an automated rating of the characteristics of the visuals offaces; generating a data set of visual modifications of a faceachievable by cosmetic and/or medical treatments and store the data setin a database; taking a standardized visual of the face of the person;inputting at least one desired characteristic of the face of the personto be changed; using the artificial intelligence to analyse the visualof the person's face and to generate a data set of modifications basedon the selected desired characteristic(s) and modifications achievableby at least one cosmetic and/or medical treatment; modifying the visualof the face of the person based on the data set of modifications andgenerate the computer-modified visual of the desired face of the personwith the modification of the face achievable by the least one proposedcosmetic and/or medical treatment; and displaying the computer-modifiedvisual of the desired face of the person.
 2. Method according to claim1, wherein the method comprises the following further steps: using atoggle mode to alternatively show the taken standardized visual of theface of the person and the computer-modified visual of the desired faceof the person, or using a marking mode to mark the areas of the face ofthe person modified by the data set of modifications as an overlay tothe displayed computer-modified visual of the desired face of theperson.
 3. The method according to claim 1, wherein the method comprisesthe following further steps: displaying all cosmetic and/or medicaltreatments stored in the data set; selecting only one or more of thecosmetic and/or medical treatments displayed; and using only selectedcosmetic and/or medical treatments for the generation of the data set ofmodifications.
 4. The method according to claim 1, wherein the methodcomprises the following further steps: displaying all characteristicsstored in the database; and selecting two or more of the characteristicsdisplayed as desired characteristic to be used by the artificialintelligence to generate the data set of modifications.
 5. The methodaccording to claim 1, wherein the method comprises the following furtherstep: displaying those cosmetic and/or medical treatments which, wereselected by the artificial intelligence to be used to generate the dataset of modifications.
 6. The method according to claim 1, wherein themethod comprises the following further steps: using the artificialintelligence to analyse the visual of the person's face to provide anautomated rating of the characteristics of the visual of the person'sface; and displaying the characteristics of the visual of the person'sface.
 7. The method according to claim 1, wherein the method comprisesthe following further steps: using the artificial intelligence toautomatically identify an area with wrinkles in the visual of theperson's face and to automatically soft focus this identified area incase the data set of modifications includes such information to modifythe visual of the face of the person to generate the computer-modifiedvisual of the desired face of the person.
 8. A system to display acomputer-modified visual of a desired face of a person with a mobiledevice, which mobile device comprises a display and a camera and inputmeans to input a characteristic, which mobile device is connected to aremote server which generates a data set of modifications, wherein theremote server comprises a deep learning based application to process thesteps of the method according to claim 1 to generate a database withdata sets of visuals of faces and extracted face property data thereoflinked to face characteristics and to generate a database with a dataset of visual modifications of a face achievable by cosmetic and/ormedical treatments and the deep learning based application is built togenerate the data set of modifications for at least one region of theface of the person to achieve the desired change in the characteristicof the face based on the generated databases; and the remote server orthe mobile device comprises visual data modification means to modify thevisual data of the face of the person with the data set of modificationsachievable by the at least one cosmetic and/or medical treatment, whichmodified visual data of the face are displayed with the mobile device.9. A computer program comprising instructions which, when the program isexecuted by a computer, cause the computer to carry out the followingsteps to provide a computer-modified visualization of a desired face ofa person: generating a data set of visuals of faces and extracted faceproperty data thereof linked to face characteristics data provided by arepresentative set of humans that rate the visuals of these faces abouttheir face characteristics and store the data set in a database;extracting further face property data of these visuals of faces and usethese extracted face property data together with the generated data setfor training of an artificial intelligence to enable the artificialintelligence to provide an automated rating of the characteristics ofthe visuals of faces; generating a data set of visual modifications of aface achievable by cosmetic and/or medical treatments and store the dataset in a database; receiving a standardized visual of the face of theperson from a mobile device; receiving at least one desiredcharacteristic of the face of the person to be changed input by theperson with mobile device; using the artificial intelligence to analysethe visual of the person's face and to generate a data set ofmodifications based on the selected desired characteristic(s) andmodifications achievable by at least one cosmetic and/or medicaltreatment; modifying the visual of the face of the person based on thedata set of modifications and generate the computer-modified visual ofthe face of the person with the modification of the face achievable bythe at least one proposed cosmetic and/or medical treatment; andtransmitting the computer-modified visual of the desired face of theperson to mobile device to display the computer-modified visual.